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ABSTRACT 

Two-factor analyses of variance with multiple 
measurements cn one factor were conducted for the five sections of 
the Test of English as a Foreign Language (TOEFL) and six language 
groups, and also for the items of each section, based on data 
collected at the first administration of this form of the test, in 
October 1969, the principal findings, that the Test x Group 
interaction and the Item x Group interactions were significant, 
indicated that some tests and some items were relatively more 
difficult for some language groups than for others. Displays of the 
Item X Group interactions were made by analyzing the item difficulty 
plots for each language group against a spaced sample of all 
candidates taking this form of the test in October 1969, A measure of 
the deviation of each item from the central tendency of the plot was 
developed, expressing the degree to which the item was especially 
difficult — or-easy — for a particular language group relative to the 
other items. Distributions of these measures are given for each of 
the five parts of TOEFL, for each of the six language groups, 
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Abstract 

Ivo-factor analyses of variance with multiple measurements on one factor 
were conducted for the five sections of the Test of English as a Foreign 
Language (TOEFli) and six language groups, and also for the iteirs of each 
section, based on data collected at the first administration of this form 
of the test, in October 1969* The principal findings, that the Test x Group 
interaction and the Item x Group interactions were significant, indicated 
that some tests and some items were relatively more difficult for sc.-e 
language groups than for others. 

Displays of the Item x Grouj, interactions were made by analyzing the 
iteiT' difficulty plots for each language group against a spaced sample of all 
cancddates taking this form of the test in October 19 ^ 9 - A- measure of the 
deviation of each item from the central tendency of the plot was developed, 
expressing the degree to which the item was especially difficult--or easy — 
for a particular language group relative to the other items . Distributions 
of these measures are given for each of the five parts of TOEFL, for each of 
the six language groups . 
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PATTERNS OF TEST AND ITEM DIFFICULTY FOR SIX FOREIGN LANGUAGE 



GROUPS ON THE TEST OF ENGLISH AS A FOREIGN LANGUAGE 

William H. Angoff emd Ami el T. Sharon 
Educational Testing Service 

Recent years have seen a significant growth in the numbers of foreign 
students enrolled in, American institutions of higher education. The 
Institute of International Education ( 1969 ) reports that the number of 
foreign students enrolled in American colleges and universities --121,362 
by their count — is more than twice that of a decade earlier. Clearly, with 
this number of students for whom the English language represents a major 
stumbling block, and with possible further growth in this number, the need 
for special language training in English is a serious and pressing problem. 

Although many universities currently offer programs to aid and advise 
foreign students in developing sufficient English language skills to cope 
with the American culture and university curriculums, much research has yet 
to be done to make these programs more effective. Many linguists have long 
felt that foreign students are considerably aided in their English language 
instruction when special emphasis is given to the specific English language 
deficiencies and patterns of errors characteristic of students of their own 
language group.. 

From the data now available (ETS, 1970 ) it is clear that the foreign 
candidates to Merican colleges who come fY*om different native language 
groups have different levels of English proficiency.^ What has not yet 
been made entirely clear is the extent to which candidates of different 
language backgrounds have more difficulty with some English language skills 
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than with others. Some information of this sort is available. King (1968), 
for example, reports that students applying to U. S . colleges from 
Afghanistan do much better in listening comprehension than in reading and 
writing in comparison with applicants from, other countries. However, far 
more information is needed on this topic of differential patterns of 
linguistic ability, not only in relation to the general skills like listen- 
ing, reading and writing, but at the more specific and detailed item level. 
It is to both levels of skill that the present study is directed. 

Instruments 



The test used in this study to assess the English proficiency of 
foreign candidates was the Test of English as a Foreign Language (TOEFL) . 
This test is administered by Educational Testing Service domestically and 
in foreign testing centers throughout the world, and foreign applicants to 
American institutions of higher education can take the examination in their 
own native countries. The test consists of five parts, each part measuring 
an English lEinguage skill thought to be important to the foreign student: 
(1) Listening Comprehension, (2) English Structure, ( 3 ) Vocabulary, ( 4 ) 
Reading Comprehension, and ( 5 ) Writing Ability. A total score, equal to 
twice the sum of the scaled scores on the five parts, is also routinely 
reported for each applicant. 



Analysis of the Sections of TOEFL 

Method 

All members of six native language groups of applicants who took TOEFL 
in October 1969 for admission to American institutions were chosen as the 
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subjects for this study. The languages were selected for study on the basis 
of two considerations: (l) the languages were to be as diverse as possible, 

representing different branches of the Indo-European family and several differ- 
ent non-Indo-European language groups; (2) they were to be represented by 
a sufficiently large number of applicants taking TOEFL. The languages 
selected, the number of subjects within each language group, and each group's 
test performance (mean and standard deviation) are given in Table 1. In 
order to provide a general baseline for evaluating the performance of the 

Insert Table 1 about here 

candidates in each language group, corresponding statistics are also 
presented in Table 1 for a "Greneral" group, a sample of 1,000 cases drawn 
at random from the general candidate group of l4,045 tested at the October 
1969 regular administration- of TOEFL ajid used for a routine-type analysis 
of the form of the test given at that administration. • 

The data collected in this study were subjected to a series of 
analyses. The first analysis was a general one, ajid sought to determine 
whether the five sections of the test retain the same order of difficulty 
across all six language groups (exclvisive of the "General” group) or 
whether there was a significant Section x Group interaction— whether some 
sections of the test were more difficult for some groups than for others 
relative to other sections of the test. For this purpose a two-factor 
analysis of variance design was used, as described in Figure 1. The first 

Insert Figure 1 about here 
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factor was the Sections of the test, which was considered fixed. The second 
factor, also considered fixed, was Groups; and since an individual could 
belong to only one of these groups, the category. Individuals, which was 
considered random, was nested within Groups. 

The linear model for the design took the following form; 



where 






^jkp ~ score of the J person in the k group on the 
t h 

p section of the test. 

H = the grand mean. 



Kj^ = the effect of the k 



th 



group. 



th 



P = the effect of the p section of the test. 

P 

till 

J^(k) = the effect of the j person in the k group. 
KPj^ = the interaction of sections and groups. 






= the interaction of sections and persons within groups. 



It is also noted that: j = l...Nj^ ; k = l...s ; and p = l...r . 

The foregoing design permitted the testing of 3 hypotheses; 

2 

1. ® • There are no differences among the six language groups 

in the mean scores across the sections of TOEFL (i.e., on total score). 

2 

2. O’ = 0 . There are no differences among the five sections of 

P 

TOEFL for all six language groups combined. 

2 

3. O’j^ = 0 . There is no interaction between the six language 
groups and the five sections of TOEFL; i.e., there are no significant 
differences among the groups with respect to their patterns of performance 
on the five sections of the test. 
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Table 2 gives a summary of the analysis of variance design. 



Insert Table 2 about here 



Results 

Table 5 gives a summary of the analyses of variance of the parts of 

Insert Table 3 about here 

TOEFL. As would be expected, with as many as 30.,600 observations (6,120 
individuals in the analysis and five observations per individual), all the 
testable F-ratios were highly significant. 

The fact that the main effect due to Sections was significant simply 
indicates that for the combined group of 6,120 candidates involved in this 
study the means on the five sections of TOEFL were not equal. However, as 
the "Percentage of Total Estimated Variance" column indicates, this source 
of variance is the smallest of all. 

The statistical significance of the effect due to Groups indicates that 
the various language groups in the study were uneqtially proficient in the 
abilities measured by the test. Reference to Table 1 indicates that the 
native speakers of German were by far the highest performers on all sections 
of TOEFL, followed in order (with some inconsistencies from one section to 
another) by speakers of Chinese, Japanese, Gujarati, Arabic, and Spanish. 
Although the language groups in this study were not intended as represen- 
tative samples of all TOEFL candidates whose native languages are German, 
Spanish, Arabic, etc., the fact that the relative order of their performance 
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is quite similar to that reported in the Interpretive Booklet for TOEFL 
(ETS, 1970) lends the present study some generality in interpretation. 

The principal finding in this analysis is the fact that the Sections 
X Groups interaction was significant, indicating that the profiles of 
performance on the five parts of TOEFL are different for the six groups of 
candidates. This may be seen graphically in Figure 2 and may also be 

Insert Figure 2 about here 

observed for any two of the six groups by plotting the pairs of means on 
the five tests for the two groups, one group on the abscissa, the other on 
the ordinate. Zero interaction would be represented by a succession of 
five points all falling on the straight line extending from lower left to 
upper right. The fact that these points do not fall in a straight line, 
but show substantial departures from the strai^t line, illvistrates the 
Section x Group interaction. 

The last two columns of Table 3 yield the further information that the 
components of variance used in this analysis as error — the source attributable 
to Individuals within Groups and the source attributable to the interaction. 
Section x Individuals within Groups— represent the largest proportion-, almost 
81^, of the total of the variance estimates. From that point of view the 
interaction. Sections x Groups, -is not at all sizeable. However, this still 
does not deny the fact that the Section x Group interaction is itself 
highly significant, representing a large component of the nonerror variance 
in the entire analysis. Whatever the proportion of the total of the variance 
estimates is represented by the Section x Group interaction, it is clear that 
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the five sections of TOEFL are quite different in difficulty for the six 
language groups, and different in different ways. 

Because the sample sizes varied so considerably ftom one language 
group to another — from ll6 for the German group to 2853 for the Chinese 
group — there remained some doubt about the adequacy of the analysis of 
variance to deal with the Section x Group interaction, in spite of the 
fact that the size of that F-ratio (236.3 for df = 20; 24,456) very likely 
precluded the possibility that its statistical significance was in question. 
Nevertheless, as a check on that possibility, a one-way multivariate analysis 
of variance was also carried out, testing for the significance of the differ- 
ences of the five-variable centroids among the six language groups. The 

results of this analysis confirmed the finding that the Section x Group 

2 

interaction was significant well beyond the .01 level. 



Analysis of Items within Sections of TOEFL 

If scores on TOEFL are to be \med successfully in diagnosis and' remedial 
training, then it would seem to be important for the teacher of English as 
a foreign language to familiarize himself with the special English language 
difficulties of students in each of the major language groups in order to 
focus the training program to their special linguistic needs . The analyses 
that are reported below are intended to support this effort by providing a 
means for studying the interaction of English language proficiencies and 
native language groups at the item level. 



Method 

For this purpose, analyses, section by section, were carried out, 
parallel to the general analysis described above. As before, a two-factor 
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analysis of variance was carried out as described in Figure 1 — except that 
here "items" replace "Sections." As before, Groups was taken to be a fixed 
effect, and Individuals within Groups was random; Items, however, unlike 
Sections above, was taken to be random. All sections were analyzed in 
identical fashion, according to a linear model of the following form: 



where 






= the score of the person in the k^^ group on 

the i^^ item! 



M = the grand mean. 

4* Vi 

I^ = the effect of the i item. 

4* Vi 

= the effect of the k^ group. 

Jj(k) = the effect of the person in the k^^ group. 

IK^j^ = the interaction of items and groups . 

IJij(k) = the interaction of items and persons within groups. 

It is also noted that: i = l...n ; j = 1...N ; k = l...s . 

iC 

Unlike the earlier design for the analysis of the parts of TOEFL, the 

foregoing analysis permitted the testing of four hypotheses: 

2 

1. ~ differences among the six language 

groups on the section of TOEFL under consideration. 

2 

2. (k) ” ^ differences among the individuals 

within the groups on the section of TOEFL under consideration. 

2 

5 . = 0 . There are no differences in the item difficulties 

in the section of TOEFL under consideration. 
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4. = 0 . There is no interaction among the six language 

groups and the items of the section of TOEFL under consideration; i.e., the 
rank order of item difficulties (on the particular section of TOEFL) is 
essentially the same for all six language groups. 

Table 4 gives a summary of the analysis of variance design for the 
items in any one of the five sections of TOEFL. Tables 5~9 give the summaries 

Insert Tables 4-9 about here 

of the analyses of variance for the items in Sections l-5> respectively. 
ResTilts 

The results shown in Tables 5-9 are quite consistent with one another. 

In every one of these tables the variances attributable to Groups, to Items, 
and to Itons x Groups are all significant far beyond the one per cent level. 
The fact that variance due to Items is significant comes as no surprise, 
since it was intended in the construction of each section to write items 
that would represent a range of item difficulties. Nor is it surprising 
that the variance due to Groups is significant. It has already been observed 
here, in Table 1 and in Table 3, that the six language groups perform differ- 
ently on the different parts of TOEFL. 

As in Table 3, the findings of principal interest in Tables 5> 6, 

8, and 9 are that the Items x Groups interaction is significant, giving 
evidence that even within the five sections of TOEFL the patterns of per- 
formance of the six language groups are systematically different. Apparently 
the items within the sections of the test are measuring sufficiently different 
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aspects of English language proficiency that they are not uniformly more 
difficult for one language group than another. 

The matter of unequal san^jle sizes across the different language 
groups represented the same kind of problem in the analyses of the Item 
X Group interactions as in the analysis of the Section x Group interaction 
reported earlier in this paper. However, because the restilts of the Section 
X Group analysis were so clearly confirmed by the multivariate analysis of 
variance, it was felt that, in view of the nature of the results of the 
Item X Group analyses, a similar confirmation was not necessary here. 



Intergroup Comparison of Item Difficulties 

The second phase of the present study involves a comparison among the 
language groups with respect to their performance on the items in each of 
the five parts of TOEFL. Since it is the purpose of this paper to describe 
a technique of analysis as much as it is to describe the results of that 
analysis, it will be useful to devote some space to that technique here. 



Method 

For each item a percent-pass figure (p-value) was computed, based on 
the performance of thie candidates in each of . the six language groups. Each 
p-value was then converted, by means of the normal probability tables, to a 
normal deviate, z , and further converted to "delta” by means of the trans- 
formation, A = 4 z + 13 i In the comparison of any two groups of examinees 

(G and G ) , the technique involves making a plot of the points, A 
X y X 

versus A , one point for each of the items under consideration. The plot 

y 

of points normally falls in an elliptical pattern extending from lower left 
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to upper right, and if the two groups studied are drawn from the same type 
of population, then the scatterplot will take the shape of a long, narrow 
ellipse, often representing a correlation as high as .98 or .99* When the 
groups are different in level, the points will still fall in a narrow 
ellipse, displaced vertically or horizontally, depending on which group is 
the more able one. Even when the groups differ in degree of dispersion, 
the points will still fall in the same type of ellipse, but in this case 
the ellipse will be tilted at an angle more or less steeply than 
depending on which group is the more dispersed. However > when the groups 
are different in type , the item difficulties will not fall in the same rank 
order for the two groups and the correlation represented by the points will 
be lower. Items falling at some distance from the elliptical swarm and 
contributing to the item x group interaction are items that are especially 
easier for one group than for the other, relative to the other items. Such 
items are deserving of special study. In the context of items like those 
in TOEFL, they may reveal detailed differences between the groups and may 
therefore lead to hypotheses regarding underlying differences in the two 
languages as reflected in the comparative abilities of the two language 
groups to deal with the . nuances of English . 

In the present study, with six language groups under consideration, 
the comparison of each group with every other group would have required 
the preparation of 15 delta plots for each of the five parts of TOEFL--a 
total of 75 plots~far too many to present in an article of limited length. 

In order to reduce sharply the number of such comparisons it was decided to 
compare each of the six language groups with a single "standard, " or General, 
group, drawn at random from the entire candidate population of about l4,000 
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taking this form of the test at its first operational administration in 

3 

October 1969 and representing candidates from many different language groups. 



Although the decision to compare each group with a single standard (General) 
group did in fact reduce the niimber of comparative A -plots, there were 
still too many to present in detail, and, accordingly, a method was so\;ight 
to summarize the significant features of the comparisons that would be made 
possible by those plots. For each elliptical plot of items the equation of 
the major axis of the ellipse was determined, and the perpendicular distance 
(D^) from each point to that line was calculated. The variance of the dis- 
tribution of these distances represents the item x group interaction and 
the distribution of these distances makes it possible to highlight those 
items that deviate sharply from the swam. The equation used for the major 
axis of the ellipse was Y = AX + B , where 



(it is recalled that the -variables, x and y , are, respectively, the delta 
values for the two groups -under consideration.) The fomula for the perpen- 




and 



B = M - AM 

y X 



dicular distance, , of each point, i , in the plot to the line is given 



as: 



AX. + B - Y. 
1 X 
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Re suits 

The distributions of the D-values in the plots for each language 
group versus the General candidate group are given in Tables 10-14, each 

Insert Tables 10 -l4 about here 

table corresponding to one of the five parts of TOEFL. In each distribution 
of the D-values in the plot of a particular language group versus the General 
group, the items with positive D-values are more difficult for the particular 
language group, relative to the other items in the ellintical svann ; the 
items with negative D-values are those that are more difficult for the 
General group, relative to the other items . This is not to say necessarily 
that the items on the positive or negative side of the zero point are more 
difficult for one group than for another. It is only that they are relatively 
more difficult for one group or the other in comparison with the other items. 
For example, in the very first distribution of Table 10, giving the D-values 
in the plot of deltas for German-speaking candidates versus the candidates 
in the General group, only one item of the 4-9 plotted was more difficult 
for the German speakers than for the General candidate group. 

In addition to the niimber of items tabulated in each distribution 
(which would equal the niimber of items in the test except when, for example, 
an item is answered correctly by everyone, or by no one, in the group, and a 
A -value and D- value cannot be calculated for it), the mean D-value is given 
(always zero), as well as the standard deviation of D-values and the corre- 
lation, r. y represented by the item plot. Both the standard deviation 
of D-values and the correlations between delta-values describe the extent 
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of the item x group interaction. In one respect the correlation between deltas 

is the preferable index, since it is a pure number, unaffected, as is , by 

the difference in the standard deviations of deltas for the two groups . The 

value of r^g can be used as an index of the similarity in the types of errors 

in English made by the two groups under consideration; and to the extent that 

their errors in English are indicative of the structure of their own language 

and the test items in English are sensitive to that structure, the value of 

r. can be used as an index of the similarities in those structures. As ex- 
ig 

pected, the different parts of TOEFL will show the different language groups 
to be similar in different degrees. It is clear that the Listening Comprehen- 
sion section of TOEFL yields interlanguage group plots that have the highest 

r. values (r. • = .9l4) of all the sections of the test, and in that sense 
ig ig 

the Listening Comprehension section represents perhaps the most homogeneous of 
all the sections of TOEFL for these six language groups. The Reading Compre- 
hension section also yields plots with high r. values (r. = .894), possibly 

because the contextual cues in a reading passage tend to equalize the influence 
of native language on comprehension. These hi^er correlations (and lower 
CTjj -values) for the Listening and Reading Comprehension sections reflect a result 
already obsejrved in Tables 5-9> that the Item x Group interactions for these 
sections are smaller than for the other three sections of the test. The lowest 
value of ( . 782 ) was found in the Writing Ability section. Therefore it 

would be assumed that the Writing Ability section — possibly in addition to English 

Structure (r. = .826) and Vocabulary “ .824) — mi^t be, generally speak- 

Ig 

ing, the most revealing sections of TOEFL for purposes of linguistic analysis. 

At a more specific level it would appear that the English Structiare section 

is most revealing for the Chinese (r. = .742) and Gujarati (r. = .752) 

Ig Ig 
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groups; the Vocabulary section is most useful for the Spanish (r. = .773) 

and possibly Chinese (.77l) groups; and the Writing Ability section most 
revealing for the German (r. = .689) and Spanish (r. = .674) groups. 
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Summary 

Two-factor analyses of variance with multiple measurements on one 
factor were conducted among the five sections of TOEFL and six language 
groups, and also among the items of each section. All sources of variance 
were found to be significant beyond the one per cent level. Of particular 
interest was the fact that the Test x Group interactions were hi^ly signifi- 
cant and that the Item x Group interactions within test were also highly 
significant, indicating that some tests and some items were relatively more 
difficult for some groups than for others. 

Further displays of the Item x Group interactions were made by analyz- 
ing the item difficulty plots for each language group against a spaced 
sample of all candidates taking this form of the test at its first formal 
administration. A measure of the deviation of each item from the central 
tendency of the plot was developed, expressing the degree to which the item 
was especially difficult— or easy— for a particular language group relative 
to the other items. Distributions of these measures are given for each of 
the five parts of TOEFL, for each of the six language groups . 

Although this paper makes no attempt to analyze the Section x Group 
and Item x Group interactions in terms of linguistic considerations, it 
does attempt to evaluate the statistical Justification for making such an 
analysis. It is then left to any investigator with competence in linguistic 
analysis to make such an analysis, if he wishes to, by examining the items 
in detail along with the statistical data on the items that will be made 
available to him on request. 
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Footnotes 

^The data referred to here are the summaries of performance of candidates 
who take the Test of English as a Foreign Language (TOEFL) as part of their 
requirements for admission to American universities. It is entirely likely 
that the differences in level as shown in the TOEFL data only reflect the 
different self-selective factors operating on these students in their native 
countries, but not the levels of performance of all English speakers in 

those countries. 

2 

The authors are indebted to F. Reid Creech for his invaluable 
assistance in carrying out the mviltivariate analysis of variance for these 
data. 

5 

The authors wish to express their appreciation to Dr. John B. Carroll 
for suggesting this solution. 
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Table 2 

Design of the Analysis of the Sections of TOEFL 



Source of Variation 


Degrees of Freedom E(MS) 


Between Individuals 


s 

S N, - 1 
k 


Groups [k] 


s - 1 Bra® + 


Indivs within Groups [j(k)] 


^ "k - = =^i(k) 


Within Individuals 


s 

(r - 1) S N 
k ^ 


Sections [p] 


r - 1 its®® + 


Sections x Groups [kp] 


(r-l)U-l) + 


Sections x Indivs wn Groups [jp(k)] 





21 
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Table 5 



Sunmary of Analysis of 


Variance for 


Sections of TOEFL 






Source of Variation 




SS 


IB 


P 


Es*t . of 
Variance 


Per Cent 
of Total 


Between Individuals 
Groups 


6,119 

5 


35,589.7*^ 

4,093.94 


818.79 


158.9* 


.1903 


10.6 


Indivs within Groups 


6,114 


31,495.80 


5.1514 




1.0505 


57.2 


Within Individuals 

Sections of the Test 


24,480 

4 


13,195.22 

808.40 


202.10 


476.1* 


.0593 


2.2 


Sections x Groups 


20 


2,005.74 


100.29 


236 . 3 * 


.1168 


6.5 


Sects X Indivs wn Groups 


24,456 


10,381.08 


0.42448 




.4245 


25.6 


Total 


30,599 


48,784.96 











*p < .01 
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Table 5 

Summary of Aneilysis of Variance 
of the Items in the Listening Comprehension Section of TOEFL 



Est. of Per Cent 



Source of Variation 




SS 




F 


Variance 


of Total 


Between Individuals 


6,119 


12 , 042.27 










Groups 


5 


688.99 


137.80 


I 9 . 9 *t 


.0031 


1.2 


Indivs within Groups 


6 ,ll 4 


11,355.28 


1.8569 


10.3* 


.0334 


13.1 


Within Individuals 


299,880 


64,007.70 










Items 


49 


8,662.04 


176.78 


979.7* 


.0341 


13.4 


Items X Groups 


245 


1,286.45 


5.2507 


29.1* 


.oo 4 o 


1.6 


Items X Indivs wn Groups 


299,586 


54,059.23 


.18044 




.1804 


70.7 



Total 305,999 76,049-97 



*p < .01 

tQuasi F ratio 




24 
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Table 6 

Summary of Analysis of Variance 
of the Items in the English Structure Section of TOEFL 



Source of Variation 


df 


SS 




F 


Est. of 
Variance 


Per Ceh 
of Totai 


Between Individuals 
Groups 


6,119 

5 


10,481.93 

2,835.88 


567.18 


48.3*+ 


.0162 


6.5 


Indivs within Groups 


6,ll4 


7,646.05 


1.2506 


7.3* 


.0270 


10.8 


Within Individuals 
Items 


238,680 

39 


47,686.95 

5,037-10 


129.16 


759.1* 


.0251 


10.0 


Items X Groups 


195 


2,077.35 


10.653 


62.6* 


.0123 


4.9 


Items X Indivs wn Grotips 


238,446 


40,572.50 


.17015 




.1702 


67.9 


Total 

< .01 


244,799 


58,168.88 











tOuasi F ratio 



Table 7 



Summary of Analysis of Variance 
of the Items in the Vocabulary Section of TOEFL 



Source of Variation 




ss 




P 


Est. of 
Variance 


Per Cent 
of Total 


Between Individuals 
Groups 


6,119 

5 


8,854.56 

647.88 


129.58 


6.9^t 


.0052 


1.2 


Indivs within Groups 


6 ,ii 4 


8,186,67 


1.5590 


7.6* 


.0291 


11.2 


Within Individuals 
Items 


258,680 

59 


51,524.58 

6,262.07 


160.57 


915.8* 


.0513 


12.1 


Items X Groups 


195 


5,454.16 


17.714 


101.0* 


.0205 


7.9 


Items X Indivs wn Groups 


2^,kk6 


4 i, 8 o 8 .i 4 


.17554 




.1755 


67.6 


Total 

< .01 

f Quasi F ratio 


24 if ,799 


60,558.93 
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Table 8 

Summary of Analysis of Variance 
of the Items in the Reading Comprehension Section of TOEFL 



Est. of Per Cent 



Soxirce. of Variation 








P 


Variance 


of Total 


Between Individuals 


6,119 


8,487.49 










Groups 


5 


458.82 


91.763 


i8.7<H- 


.0034 


1.3 


Indivs within Groups 


6,114 


8,028.67 


1.3132 


6.8* 


.0373 


14.8 


Within Individuals 


177,480 


37,007.90 










Items 


29 


2,092.20 


72.145 


372.2* 


.0140 


5.5 


Items X Groups 


i45 


549.44 


3.7892 


19.5* 


.0042 


1.7 


Items X Indivs wn Groups 


177,306 


34,366.26 


.19382 




.1938 


76.7 


Total 


183,599 


45,495.39 











^ < .01 

tQuasi F ratio 
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Table 9 

Stunmary of Analysis of Variance 
of the Items in the Writing Ability Section of TOEFL 



Est. of Per Cent 



Source of Variation 








F 


Variance 


of Toi 


Between Individuals 


6,119 


8,150.55 










Groups 


•• 5 


1,468.12 


295.62 


26.1^H- 


.0085 


5.2 


Indivs within Groups 


• 6,ii4 


6,662.21 


1.0897 


5.7* 


.0225 


. 8.8 


Within Individuals 


258,680 


52,084 .08 










Items 


59 


4,697.01 


120.44 


655.0* 


.0254 


9.1 


Items X Groiq)s 


195 


2,oi8.55 


10.551 


54.4* 


.0119 


4.6 


Items X Indivs wn Groups 


258,446 


45,568.54 


.19027 




.1905 


74.2 



Total 244,799 6o,2l4.4o 



< .01 

t Quasi F ratio 
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Table 10 

Distributions of D-Values 
for the Listening Comprehension Section 





German 


Spanish 


Arabic 


Chinese 


Japanese 


Gujarati 


1.6 - 1.7 


k 












1.1^ - 1.5 


0 






1 




1 


1.2 - 1.3 


1 




3 


0 




0 


1.0 - 1.1 


3 


1 


2 


1 


1 


2 


0.8 - 0.9 


2 


1 


1 


1 


1 


3 


0.6 - 0.7 


1 


k 


3 


k 


5 


1 


0 

• 

1 

o 

• 


1 


5 


2 


7 


k 


10 


0.2 - 0.3 


6 


6 


7 


5 


7 


3 


0.0 - 0.1 


3 


9 


7 


8 


8 


3 


-0.2 - -0.1 


7 


11 


10 


9 


13 


10 


1 

0 

• 

1 
1 

o 

• 


6 


7 


6 


5 


5 


7 


1 

0 

• 

o\ 

1 
1 

O 

• 


5 


2 


3 


3 


1 


3 


0 

• 

00 

1 
1 

o 

• 


k 


3 


.2 


2 


3 


4 


-1.0 - -0.9 


2 


1 


1 


3 


1 


1 


-1.2 - -1.1 


3 




2 


1 


0 


2 


-l.l^■ - -1.3 
No. of Items* 


1 




1 




1 








. pu " 

- 








Mean 














Std. Dev. 


0.780 


0.l^■19 


0.581 


0.536 


0.^^■6l^• 


0.582 


r . 


00 

• 


.9^0 


.906 


.931 


.9*^9 


CVJ 

0 

ON 

• 



*D-value for one item in the German group was not tabulated. 
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Table 11 



Distributions of D-Values 
for the English Structure Section 





German 


Snemish 


Arabic 


Chinese 


Jappiiese 


Guiarati 


2.0 - 2.1 


1 










1 


1.8 - 1.9 


1 










0 


1.6 - 1.7 


0 






2 




0 


1.4 - 1.5 


0 






1 


2 


2 


1.2 - 1.3 


0 




1 


1 


0 


1 


1.0 - 1.1 


1 


1 


2 


4 


3 


2 


0.8 - 0.9 


3 


1 


4 


1 


3 


2 


0.6 - 0.7 


2 


5 


2 


1 


4 


5 


0.4 - 0.5 


6 


5 


4 


3 


3 


1 


0.2 - 0.3’ 


6 


6 


7 


6 


4 


5 


0.0 - 0.1 


2 


9 


4 


3 


3 


1 


-0.2 - -0.1 


3 


4 


2 


2 


4 


5 


1 

0 

• 

1 
1 

o 

• 


4 


4 


3 


2 


3 


1 


-0.6 — 0.5 


5 


2 


3 


6 


2 


7 


• 

O 

1 

1 

GO 

• 

O 

1 


1 


1 


3 


1 


3 


0 


-1.0 - -0.9 


0 


2 


2 


2 


0 


1 


-1.2 - -1.1 


1 


1 


1 


0 


3 


0 


-1.4 - -1.3 


3 


1 


1 


3 


0 


4 


-1.6 - -1.5 


0 




0 


0 


1 


2 


-1.8 - -1.7 


1 




0 


1 


1 




-2.0 - -1.9 






1 


0 


1 




-2.2 - -2.1 








1 






No. of Items 














4 












Mean 














Std. Dev. 


0.789 


0.522 


0.716 


0.912 


0.831 


0.896 


"■ig 


.862 


.922 


.860 


.742 


.818 


.752 
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Table 12 

Distributions of D-Vedues 
for the Vocabulary Section 



German Spanish Arabic C?hinese Japanese Gu.1 jurat i 



1.8 - 
1.6 - 
1.4 - 
1.2 - 
1.0 - 
0.8 - 
0 .6 •• 
0.4 - 
0.2 - 
0.0 - 
- 0.2 - 
-0.4 - 
- 0.6 - 
- 0.8 - 
- 1.0 - 
- 1.2 - 
-1.4 - 
- 1.6 - 
- 1.8 - 
- 2.0 - 
- 2.2 - 
-2.4 - 
- 2.6 - 



1.9 


1 


2 




I'T 


0 


2 




1.5 


5 


5 




1-5 


5 


1 




1.1 


1 


5 


2 


0.9 


0 


1 


1 


0.7 


1 


0 


8 


0.5 


4 


1 


5 


0.5 


6 


2 


4 


0.1 


2 


5 


7 


-0.1 


4 


5 


5 


-0.3 


2 


6 


2 


-0.5 


2 


1 


1 


-0.7 


2 


5 


2 


-0.9 


5 


5 


1 


-1.1 


4 , 


. 2 /'■' 


1 


-1.5 


0 


1 ' 


1 


-1.5 


1 


1 


1 


-1.7 


1 


^ , 1- . 


1 


-1.9 




.0 




-2.1 




0 




-2.5 




0 




-2.5 




1 ■ 





1 

1 

2 

2 

.1 

5 

5 

5 

5 

1 

2 

2 

4 

5 
2 
0 
0 
0 
0 
0 
0 
1 



1 

0 

5 

5 

5 
9 
2 
2 

6 
5 
5 
1 
1 
0 
0 
0 
1 



1 

0 

2 

5 

2 

5 
4 

7 

2 

6 
2 
1 
2 
0 
1 
2 



No. of Items 
Mean 

Std. Dev. 



r . 
ig 



1.091 

.755 



40 

0.0 



0.676 

.858 



0.895 

.771 



0.640 

.881 



o 

ERIC 



31 



0.897 

.821 



0,676 

.878 



/ 



/ 
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Table 15 



Distributions of D-Values 
for the Reading Comprehension Section 





German 


Spanish 


Arabic 


Chinese 


Japanese 


Gujarati 


1.4 - 1.5 


1 












1.2 - 1.5 


0 


1 








1 


1.0 - 1.1 


1 


1 


1 






1 


0.8 - 0.9 


2 


1 


1 




5 


0 


0.6 - 0.7 


5 


5 


2 




0 


0 


0 

• 

1 

o 

• 


2 


2 


2 


6 


6 


4 


0.2 - 0.5 


5 


6 


5 


7 


5 


6 


0.0 - 0.1 


0 


2 


5 


5 


5 


2 


-0.2 - -0.1 


7 


2 


9 


7 


4 


6 


1 

0 

• 

1 
1 

o 

• 


2 


5 


6 


5 


5 


4 


1 

0 

• 

ON 

1 
1 

O 

• 


2 


5 


2 


5 


5 


4 


-0.8 - -0.7 


5 


5 


1 


1 


5 


2 


-1.0 - -0.9 


5 


0 










-1.2 - -1.1 




1 










-1.4 - -1.5 














-1.6 - -1.5 














No. of Items* 


















— ■ 








Mean 


























Std. Dev. 


0,658 


0.596 


0.416 


0.542 


0.457 


0.479 


r. 

ig 


.825 


.855 


.925 


.957 


.918 


.905 


*D -value 


for one 


item in the German 


group vas 


not tabulated. 
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Table l4 

Distributions of D-Values 
for the Writing Ability Section 






German 


Spanish 


Arabic 


Chinese 


Japanese 


Gujarati 


2.6 - 2.7 


1 












2.4 - 2.5 


0 












2.2- 2.5 


0 












2.0 - 2.1 


1 












H 

• 

00 

1 

H 


0 












1.6 - 1.7 


0 




1 








1.4 - 1.5 


5 


5 


0 


1 


1 


1 


1.2 - 1.5 


0 


5 


2 


0 


1 


0 


1.0 - 1.1 


2 


1 , 


0 


5 


2 


4 


0 

• 

00 

1 

o 

• 

VO 


1 


1 


0 


1 


0 


0 


0.6 - 0.7 


4 


2 


2 


6 


5 


5 


0.4 - 0.5 


2 


4 


7 


2 


2 


4 


0.2 - 0.5 


2 


5 


7 


7 


6 


4 


0.0 - 0.1 


2 


4 


2 


5 


5 


5 


-0.2 - -0.1 


6 


1 , 


4 


5 


2 


5 


1 

0 

• 

-pr 

1 

o 


1 


2 


4 


2 


12 


5 


1 

0 

• 

ON 

1 

1 

o 

• 

\J1 


2 


8 


4 


5 


4 


5 


-0.8 - -0.7 


4 


4 


2 


1 


0 


6 


-1.0 - -0.9 


5 


1 


4 


5 


0 


1 


-1.2 - -1.1 


5 


0 


1 


1 


1 


0 


-1.4 - -1.5 


2 


1 




1 


1 


0 


-1.6 - -1.5 


1 


1 




0 




1 


-1.8 - -1.7 




1 




0 






-2.0 - -1.9 








0 






-2.2 - -2.1 








1 






No. of Items 






4o 








Mean 






0.0 








Std. Dev. 


0.961 


0.852 


0.648 


0.756 


0.585 


0.656 


^ig 


.689 


.674 


.858 


.812 


.868 


.810 
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Figure Captions 

Fig. 1. Diagrammatic representation of the data for the analysis 
of variance. 

Fig. 2. Profiles of scores for the six language groups on the sections 
of TOEFL. 
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